Model learning device, method for learned model generation, program, learned model, monitoring device, and monitoring method

ABSTRACT

An image acquisition unit acquires image data in which an image of a normal monitoring target is captured. An image processing unit generates a plurality of duplicate image data pieces by performing different image processing causing a change in color tone on the image data within a range not exceeding a normal range of the monitoring target. A learning unit trains a model so as to output a value used for determining normality of the monitoring target from the image data, in which the image of the monitoring target is captured, using the plurality of duplicate image data pieces as training data.

TECHNICAL FIELD

The present invention relates to a model learning device, a method forlearned model generation, a program, a learned model, a monitoringdevice, and a monitoring method.

Priority is claimed on Japanese Patent Application No. 2017-145268,filed on Jul. 27, 2017, the content of which is incorporated herein byreference.

BACKGROUND ART

PTL 1 discloses a technique of generating a normal standard image bylearning in advance a normal-state image of a monitoring target in apast and determining normality of a monitoring target by comparing animage, which is captured by imaging the monitoring target, with thenormal standard image.

PTL 2 discloses a technique for increasing training data while savingthe effort of labeling by performing filter processing, trimmingprocessing, and rotation processing on an image as training data in acase of training a model relating to machine learning.

CITATION LIST Patent Literature

[PTL 1] Japanese Unexamined Patent Application, First Publication No.H7-78239

[PTL 2] Japanese Unexamined Patent Application, First Publication No.2016-62524

DISCLOSURE OF INVENTION Technical Problem

In the invention described in PTL 1, in order to generate the normalstandard image, it is necessary to provide a large number of images ofthe monitoring target in the normal state. On the other hand, there is ademand for quick determination of the normality of the monitoringtarget. In such a case, it may not be possible to provide a sufficientimage for learning.

An object of the present invention is to provide a model learningdevice, a learned model generation method, a program, a learned model, amonitoring device, and a monitoring method capable of appropriatelydetermining normality of the monitoring target by using a learned modeleven in a state where the volume of the training data is small.

Solution to Problem

According to a first aspect of the present invention, a model learningdevice includes: an image acquisition unit that acquires image data inwhich an image of a normal monitoring target is captured; an imageprocessing unit that generates a plurality of duplicate image datapieces by performing different image processing causing a change incolor tone on the image data within a range not exceeding a normal rangeof the monitoring target; and a learning unit that trains a model so asto output a value used for determining normality of the monitoringtarget from the image data, in which the image of the monitoring targetis captured, using the plurality of duplicate image data pieces astraining data.

According to a second aspect of the present invention, in the modellearning device of the first aspect, the image data may include athermal image having a different color tone depending on a temperatureof the monitoring target. In addition, the image processing unit maygenerate the plurality of duplicate image data pieces by performingimage processing for correcting the color tone of the image data to acolor tone corresponding to a different temperature within a changerange of an environmental temperature of the monitoring target.

According to a third aspect of the present invention, the model learningdevice of the second aspect may further include a temperatureacquisition unit that acquires temperature data indicating theenvironmental temperature of the monitoring target of when the imagedata is captured; and a correction value specification unit thatspecifies a relationship between a temperature change and a color tonecorrection value on the basis of the image data and the temperaturedata. The image processing unit may perform image processing on theimage data using the correction value specified on the basis of therelationship specified by the correction value specification unit.

According to a fourth aspect of the present invention, in the modellearning device according to any one of the first to third aspects, theimage processing unit may generate the plurality of duplicate image datapieces by performing image processing for correcting the color tone ofthe image data to a color tone corresponding to a different illuminancewithin a change range of an environmental illuminance of the monitoringtarget.

According to a fifth aspect of the present invention, the model learningdevice according to any one of the first to fourth aspects may furtherinclude a partitioning unit that generates a plurality of partitionedimage data pieces by partitioning the image data. The image processingunit may generate the plurality of duplicate image data pieces byperforming different image processing causing a change in color tone oneach of the plurality of partitioned image data pieces.

According to a sixth aspect of the present invention, the method forlearned model generation includes a step of acquiring image data inwhich an image of a normal monitoring target is captured; a step ofgenerating a plurality of duplicate image data pieces by performingdifferent image processing causing a change in color tone on the imagedata within a range not exceeding a normal range of the monitoringtarget; and a step of generating a learned model by training a model soas to output a value used for determining normality of the monitoringtarget from the image data, in which the image of the monitoring targetis captured, using the plurality of duplicate image data pieces astraining data.

According to a seventh aspect of the present invention, a program causesa computer to execute: a step of acquiring image data in which an imageof a normal monitoring target is captured; a step of generating aplurality of duplicate image data pieces by performing different imageprocessing causing a change in color tone on the image data within arange not exceeding a normal range of the monitoring target; and a stepof training a model so as to output a value used for determiningnormality of the monitoring target from the image data, in which theimage of the monitoring target is captured, using the plurality ofduplicate image data pieces as training data.

According to an eighth aspect of the present invention, a learned modelis a model which is trained so as to output a value used for determiningnormality of a monitoring target from image data, in which an image ofthe monitoring target is captured, using a plurality of duplicate imagedata pieces, which are generated by performing different imageprocessing causing a change in color tone on the image data in which theimage of the monitoring target is captured in a normal state withoutexceeding a normal range of the monitoring target, as training data. Thelearned model causes a computer to execute a step of outputting a valueused for determining normality of the monitoring target from theacquired image data.

According to a ninth aspect of the present invention, a monitoringdevice includes: an image acquisition unit that acquires captured imagedata; an inference processing unit that calculates a value used indetermining normality of a monitoring target from the image data usingthe learned model according to the eighth aspect; and a determinationunit that determines normality of the monitoring target by using thecalculated value.

According to a tenth aspect of the present invention, a monitoringmethod includes: a step of acquiring image data in which an image of amonitoring target is captured; a step of calculating a value used indetermining normality of a monitoring target from the image data usingthe learned model which is trained by the model learning deviceaccording to the ninth aspect; and a step of determining normality ofthe monitoring target by using the calculated value.

Advantageous Effects of Invention

According to at least one of the above aspects, the model learningdevice is able to appropriately determine the normality of themonitoring target on the basis of the learned model even in a statewhere the volume of the training data is small.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is a schematic diagram showing a configuration of a monitoringsystem according to a first embodiment.

FIG. 2 is a schematic block diagram showing a configuration of amonitoring device according to the first embodiment.

FIG. 3 is a flowchart showing a method for learned model generationusing the monitoring device according to the first embodiment.

FIG. 4 is a flowchart showing a normality determination method using themonitoring device according to the first embodiment.

FIG. 5 is a schematic block diagram showing a configuration of amonitoring device according to a third embodiment.

FIG. 6 is a diagram showing an example of partitioning of image dataaccording to the third embodiment.

FIG. 7 is a schematic block diagram showing a configuration of acomputer according to at least one embodiment.

BEST MODE FOR CARRYING OUT THE INVENTION Definition

The term “specify” means determining a second value at which a pluralityof values can be taken using a first value. For example, the term“specify” is defined to include calculating the second value from thefirst value, reading the second value corresponding to the first valuewith reference to a table, searching for the second value by using thefirst value as a query, and selecting the second value from a pluralityof candidates on the basis of the first value.

The term “acquire” means obtaining a new value. For example, the term“acquire” is defined to include receiving a value, receiving an input ofa value, reading a value from a table, calculating a value, andmeasuring a value.

First Embodiment

Hereinafter, embodiments will be described in detail with reference tothe drawings.

FIG. 1 is a schematic diagram showing a configuration of a monitoringsystem according to a first embodiment.

A monitoring system 10 according to the first embodiment determines thenormality of a monitoring target 100 from image data on the basis of thenearest neighbor method. The nearest neighbor method is a method ofextracting normal sample data of target data from a plurality of sampledata and determining normality on the basis of the extracted sampledata. Examples of the nearest neighbor method include the k nearestneighbor method and the local outlier method.

The monitoring system 10 includes an imaging device 200 and a monitoringdevice 300.

The imaging device 200 captures an image of the monitoring target 100and obtains visible image data, infrared image data, and thermal imagedata. The visible image data is image data obtained by measuring visiblelight reflected by the monitoring target 100. The infrared image data isimage data obtained by measuring infrared rays that are projected by theimaging device 200 and reflected by the monitoring target 100. Thethermal image data is image data obtained by measuring infrared raysemitted from the monitoring target 100. The imaging device 200 capturesimage data at regular intervals.

The monitoring device 300 determines the normality of the monitoringtarget 100 on the basis of the image data captured by the imaging device200.

FIG. 2 is a schematic block diagram showing the configuration of themonitoring device according to the first embodiment.

The monitoring device 300 includes an image acquisition unit 301, atemperature acquisition unit 302, a history storage unit 303, acorrection value specification unit 304, an image processing unit 305, alearning unit 306, a model storage unit 307, an inference processingunit 308, a determination unit 309, and an output control unit 310.

The image acquisition unit 301 acquires visible image data, infraredimage data, and thermal image data captured by the imaging device 200.

The history storage unit 303 stores image data and imaging time inassociation with each other. All the image data stored in the historystorage unit 303 is image data in which an image of the normalmonitoring target 100 is captured.

The temperature acquisition unit 302 acquires temperature dataindicating the environmental temperature of the monitoring target 100when the imaging device 200 captures image data. For example, thetemperature acquisition unit 302 acquires weather information of an areawhere the monitoring target 100 is installed through a network.

The correction value specification unit 304 specifies a correction valueused for image processing of image data. Specifically, the correctionvalue specification unit 304 specifies a gamma correction value forchanging the color tone of the visible image data in accordance with theenvironmental illuminance of the monitoring target 100 on the basis ofthe visible image data stored in the history storage unit 303. That is,the correction value specification unit 304 specifies a relationshipbetween the increase in average luminance of the visible image data andthe gamma correction value on the basis of the visible image data storedin the history storage unit 303. The visible image data has a lowerluminance and a lower contrast as the environmental illuminance islower. The correction value specification unit 304 specifies a gammacorrection value for changing the color tone of the thermal image datain accordance with the environmental temperature on the basis of thethermal image data stored in the history storage unit 303. That is, thecorrection value specification unit 304 specifies a relationship betweenthe temperature increment and the gamma correction value on the basis ofthe thermal image data stored in the history storage unit 303. Thethermal image data has a lower luminance as the temperature is lower.

The image processing unit 305 generates a plurality of duplicate imagedata pieces by performing a plurality of different image processingprocedures on the image data. Examples of the image processing includesmoothing processing, gamma correction, edge extraction, translation,rotation, and the like. The gamma correction is an example of imageprocessing that involves a change in color tone. For example, for Apieces of image data, the image processing unit 305 performs smoothingprocessing of B patterns, gamma correction of C patterns, translation ofD patterns, and rotation of E patterns so as to generate A×B×C×D×Epieces of duplicate image data. The image processing unit 305 performsdifferent image processing that involves the change in color tone withina range not exceeding the normal range of the monitoring target 100. Forexample, in a case of performing image processing of a thermal image,the image processing unit 305 performs image processing so as to changethe color tone of image data within a temperature range of a maximumtemperature to a minimum temperature during a predetermined period. Forexample, in a case of performing image processing of a visible image,the image processing unit 305 performs image processing so as to changethe color tone of the image data within an illuminance range of lightingon to off.

The learning unit 306 trains a model using the image data stored in thehistory storage unit 303 and the duplicate image data generated by theimage processing unit 305 as training data. That is, the learning unit306 causes the model storage unit 307 to store the image data, which isinput to the image acquisition unit 301, and the duplicate image data,which is generated by the image processing unit 305, as sample data forthe nearest neighbor method. A sample data group including a pluralityof duplicate image data pieces is an example of a learned model.

The model storage unit 307 stores the model trained by the learning unit306.

The inference processing unit 308 calculates an index value indicatingthe normality of the monitoring target 100 from the image data, which isinput to the image acquisition unit 301, using the model stored in themodel storage unit 307. The index value increases as the possibilitythat the monitoring target 100 is abnormal increases.

The determination unit 309 determines the normality of the monitoringtarget 100 by comparing the index value calculated by the inferenceprocessing unit 308 with a threshold value. The determination unit 309determines that the monitoring target 100 is normal in a case where theindex value is less than the threshold value. The determination unit 309determines that the monitoring target 100 is abnormal in a case wherethe index value is equal to or greater than the threshold value.

The output control unit 310 causes the output device to output thedetermination result obtained by the determination unit 309. Examples ofthe output device include a display, a printer, and a speaker.

Next, the operation of the monitoring device according to the firstembodiment will be described. The image acquisition unit 301 of themonitoring device 300 acquires image data from the imaging device 200and records the image data in the history storage unit 303 inassociation with the imaging time before creating a learned model.Thereby, the monitoring device 300 stores a plurality of image datapieces in the history storage unit 303. At this time, the monitoringdevice 300 may exclude image data by which the image of the monitoringtarget 100 cannot be reproduced due to blocked up shadows or blown outhighlights.

FIG. 3 is a flowchart showing a method for learned model generationusing the monitoring device according to the first embodiment.

The temperature acquisition unit 302 of the monitoring device 300acquires temperature data indicating the environmental temperature ofthe monitoring target 100 at the imaging time of the image data storedin the history storage unit 303 (step S1). The correction valuespecification unit 304 specifies the relationship between thetemperature increment and the gamma correction value on the basis of thecolor tone of the plurality of thermal image data stored in the historystorage unit 303 and the environmental temperature at the time ofcapturing the thermal image data (step S2). For example, the correctionvalue specification unit 304 obtains a gamma correction value at whichthe difference in luminance is minimized for two pieces of thermal imagedata, and further obtains an environmental temperature differencebetween the two pieces of thermal image data. Thereby, it is possible toobtain the relationship between the gamma correction value and thetemperature increment. The correction value specification unit 304specifies the relationship between the average luminance increment andthe gamma correction value on the basis of the color tone of theplurality of visible image data stored in the history storage unit 303(step S3).

On the basis of the temperature data acquired by the temperatureacquisition unit 302 or other weather data, the image processing unit305 specifies the maximum temperature and the minimum temperature in apredetermined period (for example, two months) starting from the currenttime (step S4). The image processing unit 305 performs gamma correctionon each thermal image data, which is stored in the history storage unit303, respectively using the gamma correction values corresponding to theincrements, which range from the environmental temperature relating tothe thermal image data to each temperature obtained by dividing therange from the minimum temperature to the maximum temperature into apredetermined number, thereby generating a plurality of duplicate imagedata pieces (step S3). For example, when the maximum temperature is 10°C., the minimum temperature is −10° C., and the environmentaltemperature relating to certain thermal image data is 0° C., and whenthe range from the minimum temperature to the maximum temperature aredivided into five. In this case, the image processing unit 305 performsgamma correction of thermal image data by using a gamma correction valuecorresponding to an increment of −10° C., a gamma correction valuecorresponding to an increment of −5° C., a gamma correction valuecorresponding to an increment of 5° C., and a gamma correction valuecorresponding to an increment of 10° C., thereby generating fourduplicate image data pieces.

The image processing unit 305 specifies the maximum value and theminimum value of the average luminance on the basis of the visible imagedata stored in the history storage unit 303 (step S6). The imageprocessing unit 305 performs gamma correction on each visible image datastored in the history storage unit 303, by respectively using gammacorrection values corresponding to increments, which ranges from theaverage luminance of the visible image data to each luminance obtainedby dividing the average luminance range into a predetermined number,thereby generating a plurality of duplicate image data pieces (step S7).

The image processing unit 305 further generates a plurality of duplicateimage data pieces by performing other image processing including atleast a smoothing process on each image data stored in the historystorage unit 303 and each duplicate image data (step S8).

The learning unit 300 trains a model using the image data stored in thehistory storage unit 303 and a plurality of duplicate image data piecesgenerated by the image processing unit 305 as training data (step S9),and records the learned model in the model storage unit 307 (step S10).

In a case where the learned model is stored in the model storage unit307, the monitoring device 300 performs normality determinationprocessing of the monitoring target 100 at each timing relating to acertain period. FIG. 4 is a flowchart showing a normality determinationmethod using the monitoring device according to the first embodiment.

The image acquisition unit 301 of the monitoring device 300 acquiresimage data from the imaging device 200 (step S51). Next, the imageprocessing unit 305 smoothes the acquired image data (step S52). Theinference processing unit 308 calculates the index value by inputtingthe smoothed image data to the learned model stored in the model storageunit 307 (step S53).

For example, the inference processing unit 308 performs the followingprocessing in a case of calculating the index value by the k-nearestneighbor method. The inference processing unit 308 calculates thedistance between the acquired image data and each sample dataconstituting the learned model. The inference processing unit 308specifies, as an index value, a representative value of distancesrelating to the k pieces of sample data having the shortest calculateddistance.

For example, the inference processing unit 308 performs the followingprocessing in a case of calculating the index value through the localoutlier factor method. The inference processing unit 308 calculates thedistance between the acquired image data and each sample dataconstituting the learned model. The inference processing unit 308calculates the densities of k pieces of the sample data having theshortest calculated distance. The inference processing unit 308specifies, as an index value, a value that is normalized on the basis ofthe density obtained by calculating a representative value of distancesrelating to k pieces of the sample data.

The distance between the sample data and the acquired image data, thedensity of the sample data, and the index value are examples of “a valueused for determining the normality of the monitoring target”.

Next, the determination unit 309 determines whether or not the indexvalue is less than a threshold value (step S54). If the index value isless than the threshold value (step S54: YES), the determination unit309 determines that the monitoring target 100 is normal (step S55). Incontrast, if the index value is greater than or equal to the thresholdvalue (step S54: NO), the determination unit 309 determines that themonitoring target 100 is abnormal (step S56).

Then, the output control unit 310 causes the output device to output thedetermination result of the determination unit 309 (step S57).

As described above, the monitoring device 300 according to the firstembodiment performs different image processing causing a change in colortone on the image data obtained by capturing the normal monitoringtarget within a range not exceeding the normal range of the monitoringtarget. Thereby, a plurality of duplicate image data pieces isgenerated, and the model is trained using these data pieces as trainingdata. Thereby, the monitoring device 300 is able to generate a largeamount of training data from a small amount of image data. Therefore, inthe monitoring system 10 according to the first embodiment, thenormality of the monitoring target can be appropriately determined bythe learned model even in a state where the original training data issmall. In the first embodiment, the gamma correction using differentgamma correction values is used as the different image processingcausing the change in color tone. However, the present invention is notlimited to this. For example, in other embodiments, different imageprocessing such as contrast correction and luminance correction usingdifferent correction values may be performed.

The image processing unit 305 according to the first embodimentgenerates a plurality of duplicate image data pieces by performing imageprocessing of correcting the color tone of the thermal image data to acolor tone corresponding to a different temperature within theenvironmental temperature change range of the monitoring target 100.Thereby, the image processing unit 305 is able to generate thermal imagedata indicating the state of the monitoring target 100 at anenvironmental temperature that is not actually observed. The correctionvalue specification unit 304 specifies the relationship between thetemperature change and the color tone correction value on the basis ofthe image data and the environmental temperature at the time of imaging.As a result, the monitoring device 300 is able to perform imageprocessing so as to obtain the color tone corresponding to the targettemperature.

The image processing unit 305 according to the first embodimentgenerates a plurality of duplicate image data pieces by performing imageprocessing of correcting the color tone of the visible image data to acolor tone corresponding to a different illuminance within theenvironmental illuminance change range of the monitoring target 100.Thereby, the image processing unit 305 is able to generate visible imagedata indicating the state of the monitoring target 100 in anillumination environment that is not actually observed.

Further, the monitoring device 300 according to the first embodimentlearns the normal state of the monitoring target 100 as a learned model.That is, in the monitoring device 300 according to the first embodiment,only image data captured by the normal monitoring device 300 in thenormal state is used as training data, and image data captured by theabnormal monitoring device 300 is not used. Therefore, it is notnecessary for the monitoring device 300 to attach a label indicatingwhether the image data is normal or abnormal in a case of using eachimage data as training data.

By the way, by continuing monitoring of the monitoring target 100, thenumber of image data pieces captured by the imaging device 200 graduallyincreases. Therefore, in a case where the learning unit 306appropriately updates the model stored in the model storage unit 307,the number of original image data (non-duplicate image data)constituting the learned model increases. The original image data ismore reliable as training data than the duplicate image data. Therefore,in a case of selecting sample data in the vicinity of input image data,the monitoring device 300 may form a model such that the original imagedata can be selected more easily than the duplicate image data. Forexample, in a case where the original image data is newly acquired in astate where the number of sample data reaches a predetermined number,the learning unit 306 may update the learned model by adding theoriginal image data to the sample data and deleting sample data that isduplicate image data. Further, for example, the inference processingunit 308 may easily select the original image data by multiplying thedistance of the original image data by a weight less than 1 in a case ofselecting the sample data. However, in a case where the inferenceprocessing unit 308 calculates the index value using the selected sampledata, the inference processing unit 308 calculates the index value onthe basis of the distance not multiplied by the weight.

Second Embodiment

The monitoring system 10 according to the first embodiment determinesthe normality of the monitoring target 100 from the image data on thebasis of the nearest neighbor method. In contrast, the monitoring system10 according to a second embodiment determines the normality of themonitoring target 100 from the image data on the basis of the neuralnetwork.

The monitoring device 300 according to the second embodiment differsfrom the first embodiment in terms of the model stored in the modelstorage unit 307 and the processing of the learning unit 306, theinference processing unit 308, and the determination unit 309.

The model storage unit 307 stores a neural network model including aninput layer, an intermediate layer, and an output layer. The number ofnodes in the input layer and the output layer is equal to the number ofpixels in the image data. The learned model functions as an auto encoderthat compresses image data, which is input to the input layer, andthereafter restores and outputs the image data. The image data, which isoutput by the learned model, is an example of “a value used fordetermining the normality of the monitoring target”.

The learning unit 306 trains a model using the image data stored in thehistory storage unit 303 and the duplicate image data generated by theimage processing unit 305 as training data. That is, the learning unit306 inputs training data to the input layer and the output layer of themodel stored in the model storage unit 307 and trains the weightingcoefficient and the activation function at each node of the input layerand the intermediate layer. Each of the training data is image data inwhich the monitoring target 100 in the normal state is captured. Forthis reason, the learned model is trained from the input image data soas to output image data in which the monitoring target 100 in the normalstate is captured. In other words, in a case where the image data inwhich the monitoring target 100 in an abnormal state is captured isinput to the learned model, it is expected that the learned modeloutputs image data in which the monitoring target 100 closer to thenormal state than that of the original image data is captured.

The inference processing unit 308 regenerates image data from the imagedata, which is input to the image acquisition unit 301, using thelearned model stored in the storage unit 307. Since the learned model istrained on the basis of the image data in which the monitoring target100 in the normal state is captured, as the possibility that themonitoring target 100 is abnormal increases, the difference between theinput image data and the regenerated image data increases.

The determination unit 309 determines the normality of the monitoringtarget 100 by calculating a difference between the input image data andthe image data regenerated by the inference processing unit 308 andcomparing the difference with a threshold value. The determination unit309 determines that the monitoring target 100 is normal in a case wherethe difference between the regenerated image data and the input imagedata is less than the threshold value. The determination unit 309determines that the monitoring target 100 is abnormal in a case wherethe index value is equal to or greater than the threshold value.

As described above, in a manner similar to the monitoring device 300according to the first embodiment, the monitoring device 300 accordingto the second embodiment is able to appropriately determine thenormality of the monitoring target using the learned model even in astate where the volume of the original training data is small.

The learned model according to the second embodiment outputs aregenerated image but it is not limited thereto. For example, in anotherembodiment, the difference between the input image data and the imagedata regenerated by the learned model may be output. In such a case, thedifference between the regenerated image data and the input image datais an example of “a value used for determining the normality of themonitoring target”.

By the way, the monitoring target 100 is continuously monitored, and thelearning unit 306 appropriately updates the model stored in the modelstorage unit 307, whereby a range of the maximum value to the minimumvalue of the environmental temperature relating to the image data usedfor learning of the learned model becomes wider. For example, in a casewhere learning is continued for half a year, the learned model istrained from image data in which the monitoring target 100 in summer iscaptured and is also trained from image data in which the monitoringtarget 100 in winter is captured. In such a case, for example, in a casewhere thermal image data in which the monitoring target 100 is in anoverheated state is used as an input and the image data is regeneratedin a learned model, there is a possibility that the difference betweenthe input image data and the regenerated image data is equal to or lessthan a threshold value. The reason for this is that the learned model isregenerated as image data relating to the temperature in the normalstate in summer as a result of learning using the image data in summer.Therefore, the monitoring device 300 may update the learning model suchthat the learning is constantly performed using the image data relatingto the predetermined period. For example, the learning unit 306 mayperiodically train a model by using image data of images captured in thelatest predetermined period in the image data stored in the historystorage unit 303, overwrite the old model, and record the model in themodel storage unit 307.

Third Embodiment

The monitoring system 10 according to the third embodiment outputs aportion of the monitoring target 100 where abnormality occurs.

FIG. 5 is a schematic block diagram showing a configuration of amonitoring device according to a third embodiment.

The monitoring device 300 according to the third embodiment furtherincludes a partitioning unit 311 and an abnormality specification unit312 in addition to the configurations of the first and secondembodiments.

FIG. 6 is a diagram showing an example of partitioning of image dataaccording to the third embodiment.

The partitioning unit 311 partitions the image data acquired by theimage acquisition unit 301 into a plurality of regions, and generatespartitioned image data. For example, the partitioning unit 311 generatessixteen partitioned image data pieces obtained by partitioning the imagedata into four equal parts vertically by four equal parts horizontally.The image processing unit 305 performs image processing on eachpartitioned image data so as to generate the duplicate image data.

The learning unit 306 trains a model for each of a plurality of regionspartitioned by the partitioning unit 311 using the partitioned imagedata and the duplicate image data as training data. The model storageunit 307 stores a learned model for each partitioned region.

The inference processing unit 308 calculates the index value of eachregion by inputting the partitioned image data partitioned by thepartitioning unit 311 to each corresponding learned model. Thedetermination unit 300 determines that the monitoring target 100 isabnormal in a case where the index value is equal to or greater than thethreshold value for at least one region, by comparing the index value ofeach region with a threshold value. The abnormality specification unit312 specifies a portion where abnormality occurs in the monitoringtarget 100 by specifying a region where the index value is equal to orgreater than the threshold value. The output control unit 310 causes theoutput device to output information indicating the portion specified bythe abnormality specification unit 312.

As described above, according to the third embodiment, the monitoringdevice 300 partitions image data to generate a plurality of partitionedimage data pieces and performs different image processing causing achange in color tone on each of the partitioned image data, therebygenerating a plurality of duplicate image data pieces. Thereby, themonitoring device 300 is able to specify the portion where abnormalityoccurs on the basis of image data.

Modification Example

As described above, the embodiment has been described in detail withreference to the drawings. However, the specific configuration is notlimited to that described above, and various design changes and the likecan be made.

For example, in the above-mentioned embodiment, the monitoring device300 performs model learning and inference on the basis of the model, butthe present invention is not limited thereto. For example, in anotherembodiment, the model learning device and the monitoring device 300 maybe provided separately, the model learning device may performs the modellearning, and the monitoring device 300 may perform inference based onthe model.

FIG. 7 is a schematic block diagram showing a configuration of acomputer according to at least one of the embodiments.

The computer 900 includes a CPU 901, a main storage device 902, anauxiliary storage device 903, and an interface 904.

The monitoring device 300 described above is mounted on the computer900. The operation of each processing unit described above is stored inthe auxiliary storage device 903 in the format of a program. The CPU 901reads a program from the auxiliary storage device 903, loads the programin the main storage device 902, and executes the above processing inaccordance with the program. In addition, the CPU 901 ensures a storagearea corresponding to each storage unit described above in the mainstorage device 902 or the auxiliary storage device 903 in accordancewith the program.

Examples of the auxiliary storage device 903 include a hard disk drive(HDD), a solid state drive (SSD), a magnetic disk, a magneto-opticaldisk, a compact disc read only memory (CD-ROM), a digital versatile discread only memory (DVD-ROM), semiconductor memory, and the like. Theauxiliary storage device 903 may be an internal medium directlyconnected to the bus of the computer 900 or an external medium connectedto the computer 900 through the interface 904 or a communication line.In a case where this program is transferred to the computer 900 througha communication line, the computer 900 that receives the program maydevelop the program in the main storage device 902 and execute the aboveprocessing. In at least one of the embodiments, the auxiliary storagedevice 903 is a non-transitory type storage medium.

Further, the program may be for realizing a part of the functionsdescribed above. Further, the program may be a so-called difference file(difference program) that realizes the above-mentioned function incombination with another program stored in advance in the auxiliarystorage device 903.

INDUSTRIAL APPLICABILITY

The model learning device according to the present invention is able toappropriately determine the normality of the monitoring target on thebasis of the learned model even in a state where the volume of thetraining data is small.

REFERENCE SIGNS LIST

10: monitoring system

100: monitoring target

200: imaging device

300: monitoring device

301: image acquisition unit

302: temperature acquisition unit

303: history storage unit

304: correction value specification unit

305: image processing unit

306: learning unit

307: model storage unit

308: inference processing unit

309: determination unit

310: output control unit

311: partitioning unit

312: abnormality specification unit

The invention claimed is:
 1. A model learning device comprising: animage acquisition unit that acquires image data in which an image of anormal monitoring target is captured, the image data including at leastone of visible image data, infrared image data, and thermal image data;a state amount acquisition unit that is configured to acquire anenvironmental state amount when the imaging device captures image data,the environmental state amount being one of temperature data indicatingthe environmental temperature of the monitoring target and environmentalilluminance of the monitoring target; a partitioning unit that generatesa plurality of partitioned image data pieces by partitioning the imagedata; an image processing unit that generates a plurality of duplicateimage data pieces by performing different image processing on the imagedata for each of a plurality of environmental state amounts within apredetermined range not exceeding a change range of a normal environmentstate amount of the monitoring target, the different image processingcausing a change in color tone on the image data based on a changingamount from the environmental state amount obtained when capturing theimage data; and a learning unit that trains a model so as to output avalue used for determining normality of the monitoring target from theimage data, in which the image of the monitoring target is captured,using the plurality of duplicate image data pieces as training data,wherein the image processing unit generates the plurality of duplicateimage data pieces by performing different image processing causing achange in color tone on each of the plurality of partitioned image datapieces, and wherein the learning unit updates the model so as tooverwrite an old model by performing learning using the image datarelating to a latest predetermined period.
 2. The model learning deviceaccording to claim 1, wherein the image data includes a thermal imagehaving a different color tone depending on a temperature of themonitoring target, and wherein the image processing unit generates theplurality of duplicate image data pieces by performing image processingfor correcting the color tone of the image data to a color tonecorresponding to a different temperature within a change range of anenvironmental temperature of the monitoring target.
 3. The modellearning device according to claim 2, further comprising: a temperatureacquisition unit that acquires temperature data indicating theenvironmental temperature of the monitoring target of when the imagedata is captured; and a correction value specification unit thatspecifies a relationship between a temperature change and a color tonecorrection value on the basis of the image data and the temperaturedata, wherein the image processing unit performs image processing on theimage data using the correction value specified on the basis of therelationship specified by the correction value specification unit. 4.The model learning device according to claim 1, wherein the imageprocessing unit generates the plurality of duplicate image data piecesby performing image processing for correcting the color tone of theimage data to a color tone corresponding to a different illuminancewithin a change range of an environmental illuminance of the monitoringtarget.
 5. The model learning device according to claim 1, wherein thelearning unit periodically trains the model to update the model so as tooverwrite an old model by performing learning using the image datarelating to a latest season.
 6. The model learning device according toclaim 1, wherein the learning unit trains the model, when in summer,from image data in which the monitoring target in summer is captured,and the leaning unit trains the model, when in winter, from image datain which the monitoring target in winter is captured.
 7. A method forlearned model generation comprising the steps of: acquiring image datain which an image of a normal monitoring target is captured, the imagedata including at least one of visible image data, infrared image data,and thermal image data; acquiring an environmental state amount when theimaging device captures image data, the environmental state amount beingone of temperature data indicating the environmental temperature of themonitoring target and environmental illuminance of the monitoringtarget; generating a plurality of partitioned image data pieces bypartitioning the image data; generating a plurality of duplicate imagedata pieces by performing different image processing on the image datafor each of a plurality of environmental state amounts within apredetermined range not exceeding a change range of a normal environmentstate amount of the monitoring target, the different image processingcausing a change in color tone on the image data based on a changingamount from the environmental state amount obtained when capturing theimage data; and generating a learned model by training a model so as tooutput a value used for determining normality of the monitoring targetfrom the image data, in which the image of the monitoring target iscaptured, using the plurality of duplicate image data pieces as trainingdata, wherein, in the step of generating the plurality of duplicateimage data pieces, the plurality of duplicate image data pieces aregenerated by performing different image processing causing a change incolor tone on each of the plurality of partitioned image data pieces,and wherein the learned model is updated so as to overwrite an oldlearned model by performing learning using the image data relating to alatest predetermined period.
 8. A non-transitory computer-readablecomputer medium storing a program for causing a computer to execute thesteps of: acquiring image data in which an image of a normal monitoringtarget is captured, the image data including at least one of visibleimage data, infrared image data, and thermal image data; acquiring anenvironmental state amount when the imaging device captures image data,the environmental state amount being one of temperature data indicatingthe environmental temperature of the monitoring target and environmentalilluminance of the monitoring target; generating a plurality ofpartitioned image data pieces by partitioning the image data; generatinga plurality of duplicate image data pieces by performing different imageprocessing on the image data for each of a plurality of environmentalstate amounts within a predetermined range not exceeding a change rangeof a normal environment state amount of the monitoring target, thedifferent image processing causing a change in color tone on the imagedata based on a changing amount from the environmental state amountobtained when capturing the image data; and training a model so as tooutput a value used for determining normality of the monitoring targetfrom the image data, in which the image of the monitoring target iscaptured, using the plurality of duplicate image data pieces as trainingdata, wherein, in the step of generating the plurality of duplicateimage data pieces, the plurality of duplicate image data pieces aregenerated by performing different image processing causing a change incolor tone on each of the plurality of partitioned image data pieces,and wherein in the step of training the model, the model is updated soas to overwrite an old model by performing learning using the image datarelating to a latest predetermined period.
 9. A non-transitorycomputer-readable computer medium storing a learned model which istrained so as to output a value used for determining normality of amonitoring target from image data, in which an image of the monitoringtarget is captured, the image data including at least one of visibleimage data, infrared image data, and thermal image data, using aplurality of duplicate image data pieces generated by performingdifferent image processing on the image data for each of a plurality ofenvironmental state amounts, an environmental state amount being one oftemperature data indicating the environmental temperature of themonitoring target and environmental illuminance of the monitoring targetwhen the imaging device captures image data, within a predeterminedrange not exceeding a change range of a normal environment state amountof the monitoring target, as training data, wherein the different imageprocessing causes a change in color tone on the image data based on achanging amount from the environmental state amount obtained whencapturing the image data, wherein the image data is partitioned togenerate a plurality of partitioned image data pieces, wherein theplurality of duplicate image data pieces are generated by performingdifferent image processing causing a change in color tone on each of theplurality of partitioned image data pieces, and wherein the learnedmodel is updated so as to overwrite an old learned model by performinglearning using the image data relating to a latest predetermined period,the learned model for causing a computer to execute the step of:outputting a value used for determining normality of the monitoringtarget from the acquired image data.
 10. A monitoring device comprising:an image acquisition unit that acquires captured image data, the imagedata including at least one of visible image data, infrared image data,and thermal image data; an inference processing unit that calculates avalue used in determining normality of a monitoring target from theimage data using the learned model according to claim 9; and adetermination unit that determines normality of the monitoring target byusing the calculated value.
 11. A monitoring method comprising the stepsof: acquiring a first image data in which an image of a normalmonitoring target is captured; acquiring an environmental state amountwhen the imaging device captures the first image data, the environmentalstate amount being one of temperature data indicating the environmentaltemperature of the monitoring target and environmental illuminance ofthe monitoring target; generating a plurality of duplicate image datapieces by performing different image processing on the first image datafor each of a plurality of environmental state amounts within apredetermined range not exceeding a change range of a normal environmentstate amount of the monitoring target, the different image processingcausing a change in color tone on the first image data based on achanging amount from the environmental state amount obtained whencapturing the first image data; and generating a learned model bytraining a model so as to output a value used for determining normalityof the monitoring target from image data, in which the image of themonitoring target is captured, using the plurality of duplicate imagedata pieces as training data, acquiring a second image data in which animage of the monitoring target is captured; calculating a value used indetermining normality of the monitoring target from the second imagedata using the learned model; and determining normality of themonitoring target by using the calculated value.